When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty

One of challenges artificial intelligence as a whole is robustness. Many issues such adversarial examples, out distribution performance, Clever Hans phenomena, and the wider areas AI evaluation explainable AI, have to do with following question: Did system fail because it hard instance or something else? In this paper we address question generic method for estimating IRT-based difficulty wide range domains covering several areas, from supervised feature-based classification automated reasoning. We show how estimate systematically using off-the-shelf machine learning regression models. illustrate usefulness estimation applications.